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ABSTRACT 

Output associated with the research and graduate 
training of university departments and individual faculty members is 
examined in the present paper. Convergence among measures of output 
is evaluated and the necessity for considering academic area when 
developing performance criteria is investigated- In addition, 2 new 
measures of scholarly output — one of publication quality and the 
other of graduate students’ first-job quality — are described, and 
their validity is evaluated in terms of their relationships to other 
output measures, (Author) 
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CONVERGENCE AMONG ACADEMIC OUTPUTS AS 
A FUNCTION OF ACADEMIC AREA 
Anthony Biglan, Gerald R. Oncken, and Fred E. Fiedler 
University of Washington 

Abstract 

Output associated with the research and graduate training of university 
departments and individual faculty members is examined in the present paper. 
Convergence among measures of output is evaluated and the necessity for 
considering academic area when developing performance criteria is investigated. 
In addition, two new measures of scholarly output — one of publication quality 
and the other of graduate students’ first job quality — are described, and 
their validity is evaluated in terms of their relationships to other output 
measures. 
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CONVERGENCE AMONG ACADEMIC OUTPUTS AS 
A FUNCTION OF ACADEMIC AREA^ 

Anthony Blglan, Gerald R. Oncken, and Fred E. Fiedler 
University of Washington 

One of the major problems confronting academic institutions is the 
evaluation of research and teaching output of individual faculty members 
and academic departments. Much evaluation is currently based arbitrarily 
on the number of journal article publications and the like. Clearly, 
academia would be better served if performance criteria were based upon 
knowledge of the types of output that are appropriate for different academic 
areas and the relationships among different types of academic outputs. 

F 

The convergence among measures of academic output is of obvious 
importance for developing such criteria.. To the extent that measures do 
converge, one method of evaluation will se,rve as well as another. However, 
in the absence of agreement among output measures it will be necessary 
to develop a multivariate approach to the problem of academic evaluation. 
Convergence among output measures has most frequently been examined in 
terms of the relationship between publication quantity and quality. 

Cole and Cole (1967) studied this problem for physicists. They based 
their index of publication quality on the number of citations of the 
scholars’ publications. This measure correlated .72 with quantity of 
publications. Clark (1957) did a similar study for psychologists in 
which he found a correlation of .45 between number of citations and the 
total number of the psychologists' publications. The correlation was 
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,36 when only the publications for a fovr~year period vere considered. 
Whether these relationships hold for other academic areas and other 
measures of publication quality is examined in the present paper. 

It is also important to ask whether quantity and quality of output 
are related in the training of graduate students. No relevant studies 
appear to be available for this question. In the present paper , the 
relationship is examined between the quantity of dissertations sponsored 
by the scholar (or, as a departmental index, the total number of 
dissertations completed in the department) and a measure of the quality 
of the first jobs which graduate students obtain, A third aspect of 
convergence is also investigated in this paper: the degree to which 

measures of research output are related to measures of the output of 
graduate .training. 

Most studies of scholarly productivity have been done in the natural 
sciences (Cole it Cole, 1967; Bayer & Folger, 1966) or psychology (Clark, 
1957). It is possible that the degree of convergence among various 
measures of scholarly output is not the same in all areas. If it is 
not the same, it would be necessary to combine criteria differently 
depending upon the area. In addition, the level and significance of each 
kind of output may differ from one type of field to another. This 
situation would necessitate differential weighting of outputs in different 
areas. Both of these possibilities are examined below. 

Quality of Research and Graduate Training 
The present paper reports a portion of the results of an extensive 
research project on the organization of university departments. As 
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part of that project two measures of the quality of output were developed. 



1. Quality of Journal Articles . A Quality of Journals Index (QJI) 
was developed as a measure of the quality of each Individual's research 
output. Because it would have been prohibitively costly to obtain 
ratings of the quality of each journal article published by each faculty 
member, the QJI was based upon ratings of the quality of the journals 
in which his articles appeared. The validity of this procedure rests 
upon two assumptions. First, it was assumed that each journal has some 
minimum editorial standards for accepting articles. Second, it was 
assumed that this minimum standard varies from journal to journal. Given 
these assumptions, journal article quality is indicated by the quality 
of journals in which the articles were published. 

To our knowledge, the QJI represents a novel approach to measuring 
publication quality. As indicated above, the most common measure of 
publication quality has been the citation index which reflects the extent 
to which the scholar's published works have been cited by others in the 
field. In thtir absence, the cost of developing citation indices becomes 
prohibitive. On the other hand, the QJI has the drawback that it only 
takes journal article publications into account. Nevertheless, this 
approach to measuring quality of research output may provide a worthwhile 
complement to the citation index approach. 

^ 2. Quality of First Jobs . One method of assessing the quality of 
graduate education in a department is to evaluate the Ph.D.s who graduate 
from it. A "First Job Index"(FJI) was based on ratings of the quality of 
first jobs obtained by graduating Ph.D.s. Development of the FJI, 
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described more fully by Fiedler and Biglan (1969), rests on the assumption 



that highly qualified Ph.D.s will seek jobs in better institutions than 
will less qualified Ph.D.s. Furthermore, it is assumed that the more 
distinguished institutions will select and will be able to attract more 
qualified candidates. Thus, an index based on the rated quality of first 
jobs obtained by graduating Ph.D.s should reflect the quality of graduate 
education provided by the department as well as the quality of students 
the department i*. able to attract. 

Methods 

The Setting 

This study was conducted at the University of Illinois, Urbana campus, 
starting in the spring of 1968. Thfj University of Illinois is a large, 
state— supported, land grant institution which places heavy emphasis on 
research and graduate training. 

The Sample 

The sample consisted of faculty members with rank of instructor or 
above from each of 44 academic departments from the University of Illinois. 
Because graduate education was a major focus of the project, only Ph.D.- 
granting departments were Included for study. 

Measurement of Outputs 

Quantity of publications . Each year the University of Illinois 
releases a pamphlet entitled, Publications of the Faculty . This pamphlet 
lists for each faculty member all of his publications for the previous 
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year. For each faculty member in our sample, the numbers of each of the 
following types of publications over the five-year period from 1964- 
1968 were tallied: textbooks, monographs, journal articles, and technical 

reports. In addition to publications, the number of doctoral dissertations 
sponsored by each faculty member for the years 1964-1968 was obtained from 
the pamphlet . 

To derive departmental level maascres of publication output , the 
total number of each of the above types of publications for the years 1964- 
1968 was computed for each department. These totals ware then divided 
by the number of faculty members in the department, resulting in per 
capita measures of departmental publications output. 

Quality of Jotirnals Index . To obtain journal quality ratings, lists 
were first compiled of all journals in which faculty members from each 
department had published articles in the years 1964 through 1968. Then 
the executive officer of each department was contacted and asked to 
recommend five or six individuals from hla department who could serve as 
judges of the quality of journals in their area. 

Each faculty judge was sent, a list of all journals in which scholars 
from his department had published articles from 1964-1968, along with 
a rating form. He was requested to rate the quality of each journal on 
a five-point scale ranging from "excellent" to "poor." A space was 
provided for the judge to indicate that he was unacquainted with the 
journal. After a telephone follow-up, a 91 per cent return rate was 
obtained on this questionnaire. 

A quality score for each journal was computed simply as the average 
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rating of that journal. To compute the QJI score for a given individual, 



each journal in which he had published each article over the five-year 
period was noted and the quality score for that journal was recorded. 

The individual QJI was then the average of the quality scores associated 
with his publications. A number of publications appeared in highly 
obscure journals and for these no ratings were obtained. The number of 
articles published in these unrated journals, turned out to be only ten 
percent of all the articles published, and these articles were, therefore, 
dropped from further calculation. A departmental QJI was computed as 
the average rated quality of all journal articles published from 1964 
through 1968 by faculty members in the department. 

The reliability of the QJI was assessed by intercorrelating the 
ratings of each pair of judges within each department. The median 
interjudge correlation across all departments was ,484, indicating that 
agreement among judges about the quality of journals was relatively low. 

In addition to judging the quality of journals, the judges were . 
asked to rate the degree of contribution of each of nine publication 
media to the expansion of knowledge in their respective fields. The 
nine media were anthologies , dissertations , monographs , patents , popular 
press, presentations at professional meetings, professional journals, 
public performances or exhibitions, and textbooks. The degree of 
contribution of each medium was rated on a five-point scale ranging from 
"contributes very much" to "detrimental." 

First Job Index . Lists of the first jobs obtained by graduating 
Ph.D.s for the years 1964-1967 were obtained from each department. Only 
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jobs obtained by male graduates in the United States were utilized. Lists 
of jobs were sent to five or six faculty members from each department 
who had been nominated by the departmental executive officer. Names of 
the students who had obtained the jobs were not Included with the lists. 
Judges were requested to rate each job as to its desirability as a first 
position for a graduating Ph.D. from his department. Ratings were made 
on an eight-point scale ranging from "very desirable" to "very undesirable." 
The average Interjudge reliability of these ratings was .58. 

A second reliabllllity measure is the split-half reliability of the 
index, i.e., a split-half correlation of items within a departmental list. 
Thus, we obtain an average first job index based on the odd items of the 
list and an average index based on the even items. The correlation over 
all departments, corrected for length, was .56, and for departments with 
at least 14 listed positions, this correlation was .72. 

The First Job Index (FJI) for a given faculty member was taken as 
the average rated quality of jobs obtained by students for whom he had 
served as dissertation sponsor. Faculty members who had sponsored no 
doctoral students received no FJI score. Departmental FJI scores were 
computed as the average rated job quality for all graduating Ph.D.s for 
the years 1964-1967. 

An obvious problem with this Index is the possibility that faculty 
members at the University of Illinois might have a tendency to rate the 
quality of jobs obtained by their own alumni as being higher than would 
"unbiased" raters. To check for this possibility, the lists of jobs from 
six of the departments in the sample were rated by six faculty nembet's in 
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each of the corresponding six departments at the University of Washington. 



jt tests indicated that the average ratings made by University of 
Washington faculty members did not differ significantly from the ratings 
made at the University of Illinois. Thus, it appears that a '*bias" 
was not operating in the ratings made at Illinois . 

In addition » for each of the six departments, the ratings made at 
the two institutions were correlated across jobs within each department. 
The median of the six correlations was .75, indicating that the First 
Job Index has satisfactory interinstitutional reliability. 

American Council on Education Rankings of Faculty Quality . Twenty- 
four of the departments in our sample were in areas which had been 
included in Cart ter *s (1966) study of the quality of departments throughout 
. the United States xdiich was conducted for the American Council on 
Education. As a result, data x^ere available for each of twenty-four 
departments on the quality of its graduate faculty as compared with 
departments in the same field at other universities. The report grouped 
departments in each field according to whether they were "distinguished," 
"strong," "good," or "adequate plus." Moreover, a department might not 
have been highly enough rated to fall into any of these categories. These 
evaluations of departments were quantified for use in the present study 
by assigning a score of 5 to "distinguished" departments, 4 to "strong" 
departments, etc. 

Classification of Academic Areas 

Biglan (1971) has presented an analysis of the task characteristics 
of academic areas vrhich is useful for the present study. He performed 
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multi-dimensional scaling of scholars’ ratings of the similarities among 
36 different academic task areas. Three orthogonal dimensions were 
defined by that analysis: 1) a "hard-soft" dimension distinguishing 

areas high in objectivity such as chemistry, engineering, and botany from 
less objective areas such as psychology, English, and education; 2) a 
"pure-applied" dimension distinguishing applied areas such as agriculture, 
education, and engineering from pure areas such as English, physics, 
and psychology; and 3) a life system dimension which distinguished areas 
such as education, agriculture, and the life sciences from areas such as 
physics, English and economics. On the basis of this analysis, depart- 
ments in the present sample can be classified into one of eight cells 
of a 2 X 2 X 2 design by dichotomizing on each dimension. Table 1 
presents such a breakdown. 

Insert Table 1 about here 



Data Analysis 

Relationship of academic area to outputs . To analyze the relation- 
ship between academic area and outputs, a 2’X 2 X 2 analysis of variance 
was performed on each of the individual outputs, based upon the hard-soft, 
pure-applied, and life system classification. This same analysis was also 
applied to the ratings of contribution of various scholarly media. 

Relationships among outputs . Relationships among individual level 
outputs were assessed by intercorrelating all output measures. This was 
first done for the entire sample of scholars. It was also considered 
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Clustering of Academic Areas in Three Dimensions 
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likely that the relationships among these output measures differ depending 
on academic area. Therefore, the sample was divided into quadrants on 
the basis of the position of the scholar's area on the hard-soft and 
pure-applied dimensions, since these two dimensions were the most 
important for distinguishing areas. Then correlations among output 
variables were computed for each sample. In addition, department level 
output meijs-ures were intercorr elated over 4A departments. 

Results and Discussion 

Relationships between Academic Area and Scholarly Output 

Table 2 presents results of analyses of variance for relationships 
between academic area and scholarly outputs. (Only results significant 
at '^.01 are shown). As indicated above, areas were categorized 
according to whether they were hard or soft, pure or applied, and 
concerned with life systems or not. The quantity of monographs and 
journal articles produced by scholars depends on whether they are in 
hard or soft areas. In hard, areas scholars produce significantly more 
journal articles and significantly fewer monographs than do their 
colleagues in soft areas. 



Insert Table 2 about here 



Scholars in applied areas produce significantly laore technical reports 
than those in pure areas. This is undoubtedly because those in applied 
areas need to communicate the details of their work to a fairly small and 
sophisticated group of consumers and the technical report represents an 
ideal method for such communication. The rated quality of graduate 
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Table 2 



Analysis of Variance for Relationships Between Academic 
Task Characteristics and Scholarly Output 
(Only significant effects <.01) are reported. 
Publication data are from the years 1964-1968.)* 



Number of Monographs 

Source H 

A Hard (.08) - Soft (.28) 1/473 14.54 

Number of Journal Articles 

A Hard (6.21) - Soft (2.72) 1/473 25.31 

. 'Quality of Journal Articles 
AC Interaction^' 1/311 8.22 

Number of Technical Reports 

Tl'''j^ure (.16) - Applied (.46) 1/473 6.64 

Number of Dissertations Sponsored 
(No significant effects) 

Graduate Students* First Job Quality 
B Pure (4.85) -Applied (5.82) 1/75 10.30 



% Variance 
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*Factor A = Hard-Soft 

B = Pure-Applied 

C = Life system -r Non-life system 
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students' first jobs is also higher in applied than in pure areas. This 
may be because applied area graduate students obtained better jobs but 
it could also be because a rating bias exists which causes faculty members 
in applied areas to rate first jobs' of their graduates more highly. 

There was a significant interaction between the hard-soft and life 
system factors as they are related to quality of journals. Hard » non- 
life system areas such as physics and chemistry had journal articles of 
higher quality than any of the other areas. Scholars in these areas at 
Illinois may have produced better research (and these are considered to 
be outstanding departments) or they may simply have evaluated journal 
articles per se more highly. 

Table 3 presents results of analyses of variance for relationships 
between academic area and the rated contribution of various scholarly 
media. Only four results were significant at the .01 level. Compared 
to scholars in soft areas, those in hard areas rate the contribution of 
both patents and presentations at professional meetings more highly. 

Also, scholars in applied areas consider presentations at professional 
meetings to be a greater contribution to the field than do scholars in 
pure areas. Finally, a significant interaction between the hard-soft 
and life system factors on the contribution of the popular press indicated 
that scholars in soft, non-life system areas such as accountancy and 
English rate the popular press as a greater contribution to their field 
thcin do scholars in other areas. 



Insert Table 3 about here 



TABLE 3 



Analyses of Variance for Relationships between Acadesilc Area 
and Rated Contribution of Scholarly Media 
(Only results significant at 2 . *01 level are shown.) 



1. Anthology 

Source OT 

(No significant effects) 

2a Contribution of Dissertations 
(No significant effects) 

3. Monographs 
(No significant effects) 

4. Patent 

A Hard (2.78) - Soft (2.05) 1/144 23.61 

5. Popular Press 

AC Interaction 1/173 11.86 

6. Presentation at Professional Meeting 

A Hard (4.22) - Soft (3.62) 1/175 19.93 

B Pure (3.74) - Applied (4.09) 1/175 6.90 

7 . Professional Journal 
(No slgnlflccint effects) 

8 . Public Performance or Exhibition 
(No significant effects) 

9. Textbook 
(No significant effects) 



% Variance 
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Taken together, the results, for the level of scholarly output and 



contribution of scholarly media point to the necessity for considering 
academic area when developing individual and departmental performance 
criteria. The same standards are not appropriate to all fields. The 
relative weight given to monographs and journal articles when evaluating 
a scholar's performance should depend on whether the area is hard or 
soft. A similar implication follows from the results for the contributions 
of patents and presentations at professional meetings ; these are 
considered greater contributions in hard areas than they are in soft areas. 
The technical report has apparently not been examined in previous studies 
of scholarly output (most of which have studied pure areas). However, 
the present study Indicates that technical reports are a more important 
output for applied areas than for pure areas, and that it would be a 
mistake to ignore them in setting up performance criteria in applied areas. 
Similarly, presentations at professional meetings deserve greater 
consideration when evaluating scholars in applied areas than when 
evaluating pure area scholars, since they are considered a greater 
contribution to tlie field in the former areas. 

Relationships Among Outputs at the Individual Level 

Table 4 presents correlations among measures of scholarly output 
taken across scholars in all academic areas. Due to the large Ns, most 
of these correlations are significantly different from zero. However, 
only five correlations are as great as .20. The number of monographs 
published by scholars is related to their rate of publication of textbooks 
(.29), journal articles (.21), and technical reports (.25). In turn. 
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the rate of journal article publication is related to the number of 
dissertations sponsored (.41) and the number of technical reports (.20). 
None of the correlations between the journal quality measure and other 
outputs was greater than .09. For the measure of first job quality, the 
greatest correlation was only .19 (with number of technical reports). 



Insert Table 4 about here 

These correlations show that there is slight convergence among the 
various measures of research output; number of monographs, number of 
journal articles, quality of journal articles, and number of technical 
reports. Of particular interest in this respect is the correlation of 
-.09 between number of journal articles and the journal quality measure. 
This result is In contrast to the typical finding of moderate to strong 
relationships between quantity of publications and measures of quality 
(cf. Cole & Cole, 1967). Similarly, we might expect that the number 
of dissertations a scholar sponsors Is related to the quality of his 
graduate students' first jobs, if for no other reason than that graduate 
students seek out the scholars who, they feel, can get them good jobs. 
However, this correlation is only .08. 

Correlations among these same variables were computed separately 
for each of four sets of academic areas: (1) hard -pure, (2) soft-pure, 

(3) hard-applied, and (4) soft-applied. These correlations are presented 
in Table 5. The correlations for the hard-pure areas, such as chemistry 
or physics, are essentially the same as the correlations over all areas. 
Especially interesting is the correlation of -.07 between journal quantity 
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N=559 except for correlations with journal quality, N=319; 
for the correlations with first job quality, N=83. N=65 for 

the correlations between first job quality and journal 
article quality. 



14 



Blglan 



and quality in hard-pure areas . It is in these areas that previous 
investigations have shown positive relationships between the quality of 
scholars’ journal articles (as measured by citation rates) and their rate 
of journal article publication. 

Insert Table 5 about here 

The second set of correlations is for soft-pure areas such as 
English and political science. The only distinct difference between this 
set of correlations and the ones presented above is that the number of 
technical reports which scholars publish is highly related to their rate 
of journal article publication. 

The third set of correlations presented in Table 5 is for hard-applied 
areas such as engineering and agriculture. In contrast to the correlations 
for hard-pure and soft-pure areas, as well as for all areas taken together 
in hard-applied areas, such as engineering, first job quality is negatively 
and significantly related to the number of journal articles published (-.57) 
and the number of dissertations sponsored (-.49). Moreover, in these 
areas the number of journal articles which scholars publish is 
significantly related to every other variable except the quality of journal 
articles. Once again, this is in striking contrast to previous studies 
which have shown positive correlations between the quantity and quality 
of scholars' publications. 

The final set of correlations in Table 5 is for soft-applied areas 
such as education and finance. In these areas the various measures of 
research output are related in about the same manner- as in other areas. 
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However, the measure of first job quality is more strongly related to 
other variables in soft-applied areas than in other areas. It is 
positively and significantly related to both the number of technical 
reports produced by scholars (. 58 ) and the quality of their journal 
articles (. 54 ). 

Regardless of the academic area considered, there appears to be at 
best only moderate convergence among measures of research productivity. 
Some of the strongest relationships are between number of dissertations 
sponsored and other measures of publication quantity. However, this 
measure is as much an index of effectiveness in graduate training as 
it is a measure of the sponsor's research productivity. Convergence 
among publication outputs is smallest in hard-pure areas such as physics 
and zoology. These results show that it is a mistake to evaluate scholars 
in terms of a single criterion such as journal article publication. 

The scholar who produce's many publications of one type may not produce 
publications of another type. One reassuring note in these findings, 
however, is that in all areas publication rate for journal articles is 
consistently positively related to the number of dissertations sponsored. 
This suggests that scholars who are productive researchers are more, 
rather than less, involved in graduate students* work on their 
dissertations. Thus, according to these data, research and graduate 
education do not conflict. 

Contrary to existing literature, this study indicates that the 
quality and quantity of journal article publications are unrelated. This 
is true regardless of tb,e academic area. As mentioned earlier, previous 



O 

ERIC 



^ O 



16 



Big Ian 



research employed citation rates as a measure of journal quality. Cole 
and Cole (1967) found that citation rate correlated .72 with the number 
of publications. No data are available on the relationship between 
citation rate and the quality measure used In the present study. It 
Is possible, therefore, that the difference between the present result 
and previous findings Is due to dissimilar methods of measuring publication 
quality. 

An Intriguing alternative to this explanation Is that the quantity 

and quality of publications are not as strongly related as they were 

2 

when earlier studies were done. This could happen If Increasing 
ntxmbers of scholars were motivated to publish but their publications 

were not of high quality. Many newly trained scholars may be motivated 

♦ 

to publish more to further their own careers, than to make a contribution 
to the field. Although this argument would be difficult to support 
empirically, it is consistent with a number of existing critiques of 
contemporary scholarship. For example, Gouldner (1970) notes the 
development of a careerist orientation among sociologists in which 
prestige in society and financial success are increasingly Important 
due to the enlarged role of sociologists in society. Similarly, 

Chomsky's (1969) critique suggests that scholars in social sciences and 
history are increasingly motivated to obtain prestige and power in the 
larger social system. 

When all academic areas are considered together, the quality of 
graduate students* first jobs is unrelated to any other output- measure* 
However, in hard-applied and soft-applied areas, a number of distinct 
relationships occurred. In hard-applied areas such as engineering and 
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agriculture, first job quality is significantly negatively related to 
the number of journal articles published (-.57) and the number of 
dissertations sponsored (-.49). It was negatively, though not signifi- 
cantly related to all but one of the remaining variables. In soft- 
applied areas such as education and accountancy, first job quality is 
positively related to all outputs and significantly related to number of 
technical reports published (.58) and the quality of journals (.54). The 
negative relationships in hard— applied areas suggest that the scholar 
who is publishing journal articles and sponsoring dissertations cannot 
necessarily obtain the best jobs for his graduate students. It is likely 
that for many of the jobs in these areas, graduate students are not 
hired on the basis of their own or their sponsor’s research prominence, 
but rather on the basis of practical skills which were not Indexed in 
the present study. In soft-applied areas, however, it appears that at 
least one form of research prominence — the quality of the sponsor’s 
journal articles — is an important factor related to the graduate student’s 
first job. The fact that in soft-applied areas the remaining publication 
outputs are positively associated with first job quality supports this 
interpretation . 

Relationships Among Outputs at the Department Level 

Table 6 presents correlations among scholarly outputs at the 
department level. These relationships are important for determining the 
value of these variables as indices of departmental performance. Publi- 
cation data were divided by the number of faculty in the department in 
1967. In addition to the outputs examined at the individual level, this 
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table Included American Council on Education ratings of the quality of 
the department*s faculty (Cartter, 1966). Since a department’s reputation 
may. In part, be due to size, the correlations between total number of 
faculty In the department and Journal quality, first job quality, and 
ACE ratings were also computed. 

Insert Table 6 about here 



Similar to the findings for the Individual level, there Is only 
slight convergence among measures of research outputs when they are used 
as department criteria. The number of dissertations sponsored per 
faculty member Is significantly positively related to monographs per 
faculty member (.47). The number of technical reports per faculty member 
Is significantly positively related to journal articles per faculty member 
(.38). Also, like the individual level results, journal article quantity 
and quality are unrelated (.04). Journal quality is significantly 
negatively related to the ntmaber of technical reports per faculty member. 

The rated quality of graduate students' first jobs is negatively 

related to the number of monographs per faculty (Jr = -.48, £_ < .05) and 

1 

positively related to the number of journal articles per faculty member 
(r = .39, < .05). Thus, departments which emphasize journal article 

publication and deemphaslze monograph publication send graduate students 
on to the best jobs. And the first job measure shows adequate convergence 
with the ACE ratings of faculty quality (r = .44, < .05). Although 

there was a slight positive relationship between first job quality and 
dissertations per faculty member, the correlation was not significantly 



TABLE 6 



Correlations Among Output Variables at 
the Department Level 
(Publications are per faculty member) 





Monographs 

1 


Journal 

1 Articles 

1 

i , 


Journal 

Quality 


Technical 

Reports 


Dissertations 


First Job 
Quality 


Articles 


-.08 












Quality of 


-.03 


.04 










Journal Articles 


(A2) 


(42) 










Technical 


.01 


.38* 


-.47* 








Reports 






(42) , 








Dissertations 


o47* 


.28 


.13 


-.02 












(42) 








First Job Quality 


-.48 * 


.39 * 


.30 


.09 


.28 






(35) 


(35) 


(34) 


(35) 


(35) 




ACE Rating 


-.13 


.52* 


.51* 


.01 


.67* 


.44* 


- 


(19) 


(19) 


(18) 


(19) 


(19) 


(24) 


Total Number 






.10 






-.17 


of Faculty 






(42) 






(43) 



N=44 except wl are otherwise indicated in parentheses. 
*p < .05 
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different from zero (.28). Thus, as is the case for journal articles, 
quantity and quality appear to be unrelated for graduate training. 

The results of this study constitute strong evidence for the 
validity of the ACE ranking as an index of departmental quality. It is 
significantly positively related to all but two of the other output 
measures. Departments which are highly rated by the ACE are also higher 
in number of journal articles per faculty member (.52), number of 
dissertations per faculty member (r »= -67), journal article quality 
( r =» .51), and the quality of graduate students' first jobs (r « .44). 

The relationships with journal article quantity and quality suggest that 
the ACE ratings are meaningfully assessing research productivity and not 
just reputation. The evidence iy all the more convincing since, according 
to these data, journal article quality and quantity comprise separate 
components of research output. Similarly, the ACE ratings appear to be 
tapping both components of graduate training, namely the quantity of 
graduate students v/ho complete graduate work and their quality as 
indexed by the jobs they are able to obtain. There is, however, an 
additional relationship which should temper a positive judgment concerning 
the ACE ratings. ACE ratings correlate. .54 (N = 24, < .05) with the 

total number of faculty members in the department. This suggests that 
the department's "visibility" may be an important factor determining a 
department's high ratings on the ACE measure. 

Conclusion 

There are a number of implications of these findings which deserve 
special emphasis. 
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1. No single criterion of Individual scholarly performance seems 
possible * Regardless of academic area, such measures of research output 
as the quantity of journal articles and monographs and the quality of 
journal articles fail to converge with such measures of graduate training 
effectiveness as dissertations per faculty member and first job quality. 
Moreover, measures of research output show only moderate convergence with 
each other and the measures of graduate training effectiveness do not 
converge at all. No measures of the effectiveness of undergraduate 
teaching were available in the present study, but we may surmise on the 
basis of these dsta that undergraduate teaching performance is not related 
to other outputs. 

2. The ACE ratings of faculty quality appear to provide a good 
measure of department performance . They are related to both the measures 
of research output and the measures of graduate training effectiveness. 

The only possible drawback to this ACE evaluation is its relationship 

to department size, and this measure may, therefore, be partly a result 
of departmental visibility. 

3. Academic area should be considered in developing performance 
criteria . This study shows that the rate of scholars* journal article 
and monograph publications as well as the quality of their journal 
articles depends on whether the area is "hard” or "soft.” Moreover, the 
rate of scholars' technical report publication is greater for applied 
than for pure areas. These results suggest that the weight we attach to 
each output in evaluating the scholar should depend on the academic 
area. In addition, the relationships among outputs are not always the 
same in different areas. In hard-applied areas such as agriculture and 
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engineering, the graduate students who obtain the best jobs are not 
associated with the more productive researchers or with the scholars 
who sponsor large numbers of doctoral dissertations. On the other hand. 
In sof t“’applled areas such as education, the best jobs appear to go to 
the graduate students who are associated with the more productive 
researchers. These findings Imply that for some areas, attempts to 
enhance one kind of output may detract from other kinds of output. 

The study presents the First Job Index and the Quality of Journals 
Index as potentially useful measures for assessing two different aspects 
of department performance. While the measures are not sufficiently 
reliable at this time to be useful criteria of Individual faculty 
member performance, they appear satisfactory for evaluating the 
department's performance. 



O 

ERIC 



ou 



Biglan 



References 



22 



Bayer, A. E. , & Folger, J. Some correlates of a citation measure of 

productivity In science. SocloJjogy of Education , 1966, 39 , 381-390. 

Biglan, A. The characteristics of the tasks of academic areas. 

Seattle, VJashington: Organizational Research, Technical Report 71-23, 

University of Washington, July, 1971. 

Cartter, A. M. An Assessment of Quality In Graduate Education . 

American Council on Education, Washington, D.C., 1966. 

Chomsky, Noam. American Power and the New Mandarins . Ue\<r York: 

Vintage Books, 1969. 

Clark, K. E. America’s Psychologists , American Psychological Association, 
Washington, D.C., 1957. 

Cole, S. , & Cole, J. R. Scientific output and recognition: A study In 

the operation of the reward system In science. American Sociological 
Review , 1967, 377-390. 

Fiedler, F. E. , & Biglan, A. The First Job after Graduation as a Measure 
of Departmental Performance. Urbana, Illinois: Group Effectiveness 

Research Laboratory, Technical Report 74, University of Illinois, 
April, 1969. 

Gouldner, A. The Coming Crisis In Western Sociology . New York: Basic 





Books, 1970. 



23 



Blglan 



Fo otno tes 

1. This study was supported, in part, by a grant from the U.S. Office of 
Education, O.E. Bureau of Research No. 0-0340, Grant No. OEG-0-70, 3347 
(Fred E. Fiedler, Principal Investigator), and, in part, by the 
Office of the Executive Vice President and Provost, University of 
Illinois. 

2. The authors would like to thank Bryant Lindsey for this suggestion. 




32 



